Skip to content

database: close pgxpool on db.Close and apply pool defaults to pgxpool - #4

Open
adamdecaf wants to merge 14 commits into
stevemsmith:fix/alloydb-failover-recoveryfrom
adamdecaf:fix/alloydb-pgxpool-close-and-defaults
Open

database: close pgxpool on db.Close and apply pool defaults to pgxpool#4
adamdecaf wants to merge 14 commits into
stevemsmith:fix/alloydb-failover-recoveryfrom
adamdecaf:fix/alloydb-pgxpool-close-and-defaults

Conversation

@adamdecaf

Copy link
Copy Markdown

Summary

Follow-up fixes for moov-io#490.

  1. Close pgxpool (and AlloyDB dialer) when *sql.DB is closed
  2. Apply ConnectionsConfig defaults on pgxpool, not database/sql

See commit message for details.

bpross and others added 14 commits June 26, 2026 10:03
Postgres: NewPostgresDatabaseFromTemplate creates isolated test databases
by cloning a content-addressed template database that has already been
migrated. The template is created once per unique migration set (SHA256
of migration filenames and contents) and reused across all tests in the
process. Uses pg_advisory_lock for cross-process safety and marks the
template with IS_TEMPLATE=true, ALLOW_CONNECTIONS=false.

Spanner: NewSpannerDatabaseFromMigrations and CreateSpannerDatabaseFromMigrations
create a Spanner test database with all migrations applied in a single
batched DDL update via UpdateDatabaseDdl, instead of running each migration
file individually through golang-migrate. The spanner_schema_migrations
table is pre-populated with the latest version so subsequent RunMigrations
calls are no-ops (ErrNoChange). Falls back to normal migration if DDL
parsing fails.

Shared helpers: migrationFiles, migrationHash, and migrationVersion
utilities for walking embedded migration FS, computing content-addressed
hashes, and extracting version numbers from filenames.
Pre-existing govulncheck failure in admin.TestAdmin__AddHandler via
idna.ToASCII. Not related to the testdb changes but blocking CI.
Covers migrationFiles, migrationHash, and migrationVersion with
embedded test fixture migrations. Brings testdb package coverage
above 0% to satisfy CI coverage threshold.
Windows CI reads embedded files with CRLF line endings, causing
exact string equality assertions to fail.
Some tests (e.g. TestEnvironment in card-gateway) call
service.NewEnvironment directly without going through CreateTestDatabase.
The old CreateTestDatabase created the service database as a side effect
via openOrCreateDatabase. NewPostgresDatabaseFromTemplate now preserves
that behavior by creating the service database if it doesn't exist.

Also fix nil pointer panic by guarding testEnv.Shutdown in card-gateway.
Use a dedicated Postgres connection for advisory lock acquisition and
release so the session-level lock is not leaked across pooled connections.

Clean up Spanner databases on post-create failures and guard nil migration
filesystems before reading migrations.
Adds unit coverage for service database creation side effects,
content-addressed lock keys, and nil Spanner migration files so short CI
coverage stays above threshold.
Use the advisory lock only for the template lifecycle: check whether the
content-addressed template exists, create and migrate it if missing, then
mark it as a template before unlocking. The per-test clone happens after
unlocking so clone creation is not unnecessarily serialized.
feat(testdb): fast test database creation for Postgres and Spanner
…oyDB

During AlloyDB maintenance switchovers (< 1s downtime), services using
database/sql with pgx fail to recover because:
1. No connection pool limits are set by default, so dead connections
   persist indefinitely
2. No retry logic exists for transient connection errors

This adds:
- DefaultPostgresConnectionsConfig() with MaxLifetime=5m, MaxIdleTime=30s
  to ensure dead connections are evicted quickly after failover
- ApplyPostgresConnectionsConfig() that fills in safe defaults when
  services don't explicitly configure pool settings
- IsRetryablePostgresError() to classify transient PG/network errors
- RetryPostgres() for services to wrap critical DB operations

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Switch from sql.Open("pgx") to pgxpool.NewWithConfig() wrapped with
stdlib.OpenDBFromPool(). This gives us pgxpool's HealthCheckPeriod
(set to 5s) which proactively pings idle connections and evicts dead
ones — the most important fix for surviving AlloyDB maintenance
switchovers. The return type remains *sql.DB so no downstream changes
are needed.

Also cleans up the dialer TODO (dialer lifecycle is now tied to the
pool) and removes the unused pgx.ParseConfig import path.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
For sub-second AlloyDB switchovers, 1s health checks detect and evict
dead connections faster with negligible overhead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Collapse switch cases in IsRetryablePostgresError for readability
- Make context cancellation test more specific (assert context.Canceled
  and exact call count)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
stdlib.OpenDBFromPool does not close the underlying pool when *sql.DB is
closed, leaking connections and the health-check goroutine. Wrap the pool
connector with io.Closer so db.Close() shuts down the pgxpool and AlloyDB
dialer.

Also stop applying ConnectionsConfig via database/sql setters for Postgres:
those do not configure pgxpool, and SetMaxIdleConns(n>0) breaks
OpenDBFromPool. Map resolved defaults onto pgxpool instead so unset MaxOpen
does not silently fall back to max(4, NumCPU()).
@stevemsmith
stevemsmith force-pushed the fix/alloydb-failover-recovery branch from 6546de8 to 0019f7c Compare July 30, 2026 21:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants